Applying a facial mask to a face in a video is very different from applying a facial mask to a face in an image. In particular, an image is stationary; therefore, once the image and the facial mask are properly aligned with one another the facial mask can be placed in front of the face (or blended with the face to make the mask semi-transparent). However, in a video the face and many facial elements are in motion, making the addition of a facial mask a particularly difficult problem (background, for example, is easy to replace since it can be filmed as a stationary element such as a green screen). For example, the placement of the face, expressions on the face, the angle of the face relative to the camera, etc. may all change from frame to frame. Even if the user tries to hold still there are movements that are impossible to suppress for long periods of time, such as blinking or iris movement.
Because of this difficulty facial masks are generally applied on a frame by frame basis. Further, it is difficult to fully automate this process. For example, in one frame the whole facial mask may move during an automated process relative to the face because of movements of part of the face, such as a jaw when the actor is talking. Therefore, this process is accomplished by hiring a specialist that adds the facial mask to each frame. This allows them to control elements such as those discussed above that would cause the mask to move and would cause a disorienting effect.
Since these changes are made in a very time intensive process it is impossible to make the changes in real-time. I.e., the process can sometimes be finished weeks or months after the video is actually recorded.
Accordingly, there is a need in the art for a method that can apply facial masks to a face in real-time. Further, there is a need for the method to apply the facial mask in a manner that allows the facial mask to respond to facial movements of the user.